Skip to content

Guard builtin formatter tokenization#3580

Merged
koxudaxi merged 2 commits into
mainfrom
perf/builtin-formatter-guards
Jul 7, 2026
Merged

Guard builtin formatter tokenization#3580
koxudaxi merged 2 commits into
mainfrom
perf/builtin-formatter-guards

Conversation

@koxudaxi

@koxudaxi koxudaxi commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • Performance Improvements

    • Reduced unnecessary tokenization when lines can’t contain comment markers or multi-line strings, speeding up formatting for many inputs.
    • Added performance coverage for generating large Pydantic v2 model sets (builtin formatter vs no formatter).
  • Bug Fixes

    • Improved blank-line normalization to preserve spacing around docstrings and avoid misclassifying backslash-continued strings.
  • Tests

    • Added regression tests to ensure tokenization guards behave correctly for # comments and multi-line string detection (including newline style variations).

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2bf99b23-19cb-48de-b2fa-93cc876c48ae

📥 Commits

Reviewing files that changed from the base of the PR and between 4f92331 and 33dce84.

📒 Files selected for processing (2)
  • src/datamodel_code_generator/_builtin_formatter.py
  • tests/test_format.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/datamodel_code_generator/_builtin_formatter.py
  • tests/test_format.py

📝 Walkthrough

Walkthrough

Adds fast paths in the built-in formatter to skip tokenization when comment or multi-line-string checks are unnecessary. New regression tests cover the guards, and benchmark tests cover large Pydantic v2 model generation.

Changes

Tokenization Fast-Path Optimization

Layer / File(s) Summary
Fast-path guards in formatter helpers
src/datamodel_code_generator/_builtin_formatter.py
_has_comment_token short-circuits when no # is present; _normalize_top_level_blank_lines only tokenizes for multi-line strings when triple-quote or backslash-newline markers exist in the code.
Regression tests for tokenization guards
tests/test_format.py
Adds tests that monkeypatch tokenize.generate_tokens to fail, confirming tokenization is skipped when safe and still runs when comments or multi-line strings could exist, including docstring and backslash-continuation cases.
Performance benchmarks for large model generation
tests/main/test_performance.py
Adds benchmark tests generating 500 Pydantic v2 models using the built-in formatter and with no formatters, asserting output contains at least 500 model classes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding guards to avoid unnecessary builtin formatter tokenization.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/builtin-formatter-guards

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

📚 Docs Preview: https://pr-3580.datamodel-code-generator.pages.dev

@codspeed-hq

codspeed-hq Bot commented Jul 6, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 15.96%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 11 improved benchmarks
🆕 2 new benchmarks
⏩ 98 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime test_perf_aws_style_openapi_pydantic_v2 2.3 s 1.9 s +19.27%
WallTime test_perf_stripe_style_pydantic_v2 2.3 s 1.9 s +18.2%
WallTime test_perf_openapi_large 3.5 s 2.9 s +17.87%
WallTime test_perf_kubernetes_style_pydantic_v2 3 s 2.6 s +17.52%
WallTime test_perf_all_options_enabled 6.2 s 5.4 s +16.15%
WallTime test_perf_graphql_style_pydantic_v2 907.4 ms 782.2 ms +16.01%
WallTime test_perf_large_models_pydantic_v2 4.1 s 3.6 s +15.76%
WallTime test_perf_complex_refs 2.3 s 2 s +14.93%
WallTime test_perf_duplicate_names 1.2 s 1 s +14.18%
WallTime test_perf_deep_nested 5.8 s 5.2 s +13.15%
WallTime test_perf_multiple_files_input 3.9 s 3.4 s +12.71%
🆕 WallTime test_perf_large_models_pydantic_v2_builtin N/A 928.4 ms N/A
🆕 WallTime test_perf_large_models_pydantic_v2_noformat N/A 899.9 ms N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing perf/builtin-formatter-guards (33dce84) with main (aa7d87f)

Open in CodSpeed

Footnotes

  1. 98 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
src/datamodel_code_generator/_builtin_formatter.py (1)

1718-1725: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Guard logic is sound but only accounts for \n-terminated backslash continuations.

Per Python's lexical analysis, multi-line STRING tokens can only arise from triple-quoted literals or backslash line continuations, so the general approach here is correct. However, the check "\\\n" in code will miss a backslash continuation that uses CRLF line endings (i.e., "\\\r\n"), since the substring \ immediately followed by \n wouldn't be present when a \r sits between them. If code could ever contain CRLF line endings alongside a backslash-continued single-quoted string, the guard would incorrectly skip tokenization and string_lines would stay empty, causing blank-line normalization logic to misclassify lines inside that string as top-level code.

This is likely a non-issue in practice since generated code from this formatter is presumably always \n-terminated, but worth confirming.

🛡️ Optional defensive fix
-    if ('"""' in code) or ("'''" in code) or ("\\\n" in code):
+    if ('"""' in code) or ("'''" in code) or ("\\\n" in code) or ("\\\r\n" in code):
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/datamodel_code_generator/_builtin_formatter.py` around lines 1718 - 1725,
The guard in _normalize_top_level_blank_lines only detects backslash
continuations with LF endings, so update the multi-line STRING precheck to also
recognize CRLF backslash continuations before deciding whether to tokenize. Keep
the existing tokenization-based approach in _normalize_top_level_blank_lines,
but broaden the continuation detection so string_lines is populated correctly
when code contains "\r\n" line endings.
tests/main/test_performance.py (1)

128-158: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate test bodies — consider parametrizing.

The two tests are identical except for the formatters argument and naming. Consolidate with pytest.mark.parametrize.

♻️ Suggested consolidation
-@pytest.mark.perf
-@pytest.mark.benchmark
-def test_perf_large_models_pydantic_v2_builtin(tmp_path: Path) -> None:
-    """Performance test: Generate 500 Pydantic v2 models with the built-in formatter."""
-    output_file = tmp_path / "output.py"
-    generate(
-        input_=PERFORMANCE_DATA_PATH / "large_models.json",
-        input_file_type=InputFileType.JsonSchema,
-        output=output_file,
-        output_model_type=DataModelType.PydanticV2BaseModel,
-        formatters=[Formatter.BUILTIN],
-    )
-    content = output_file.read_text()
-    assert content.count("class Model") >= 500
-
-
-@pytest.mark.perf
-@pytest.mark.benchmark
-def test_perf_large_models_pydantic_v2_noformat(tmp_path: Path) -> None:
-    """Performance test: Generate 500 Pydantic v2 models without formatting."""
-    output_file = tmp_path / "output.py"
-    generate(
-        input_=PERFORMANCE_DATA_PATH / "large_models.json",
-        input_file_type=InputFileType.JsonSchema,
-        output=output_file,
-        output_model_type=DataModelType.PydanticV2BaseModel,
-        formatters=[],
-    )
-    content = output_file.read_text()
-    assert content.count("class Model") >= 500
+@pytest.mark.perf
+@pytest.mark.benchmark
+@pytest.mark.parametrize(
+    ("formatters", "test_id"),
+    [([Formatter.BUILTIN], "builtin"), ([], "noformat")],
+    ids=["builtin", "noformat"],
+)
+def test_perf_large_models_pydantic_v2_formatting(tmp_path: Path, formatters: list, test_id: str) -> None:
+    """Performance test: Generate 500 Pydantic v2 models with varying formatting."""
+    output_file = tmp_path / "output.py"
+    generate(
+        input_=PERFORMANCE_DATA_PATH / "large_models.json",
+        input_file_type=InputFileType.JsonSchema,
+        output=output_file,
+        output_model_type=DataModelType.PydanticV2BaseModel,
+        formatters=formatters,
+    )
+    content = output_file.read_text()
+    assert content.count("class Model") >= 500
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/main/test_performance.py` around lines 128 - 158, These two performance
tests duplicate the same generate/assert flow and only differ in the formatters
argument, so consolidate them into one parametrized pytest test. Refactor the
duplicated logic in test_perf_large_models_pydantic_v2_builtin and
test_perf_large_models_pydantic_v2_noformat using pytest.mark.parametrize over
the formatter set while keeping the existing generate call and class count
assertion intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/datamodel_code_generator/_builtin_formatter.py`:
- Around line 1718-1725: The guard in _normalize_top_level_blank_lines only
detects backslash continuations with LF endings, so update the multi-line STRING
precheck to also recognize CRLF backslash continuations before deciding whether
to tokenize. Keep the existing tokenization-based approach in
_normalize_top_level_blank_lines, but broaden the continuation detection so
string_lines is populated correctly when code contains "\r\n" line endings.

In `@tests/main/test_performance.py`:
- Around line 128-158: These two performance tests duplicate the same
generate/assert flow and only differ in the formatters argument, so consolidate
them into one parametrized pytest test. Refactor the duplicated logic in
test_perf_large_models_pydantic_v2_builtin and
test_perf_large_models_pydantic_v2_noformat using pytest.mark.parametrize over
the formatter set while keeping the existing generate call and class count
assertion intact.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3f33572a-d0ad-4f7a-8598-5fc78fcc8fa4

📥 Commits

Reviewing files that changed from the base of the PR and between aa7d87f and 4f92331.

📒 Files selected for processing (3)
  • src/datamodel_code_generator/_builtin_formatter.py
  • tests/main/test_performance.py
  • tests/test_format.py

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (aa7d87f) to head (33dce84).

Additional details and impacted files
@@            Coverage Diff            @@
##              main     #3580   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          163       163           
  Lines        35010     35033   +23     
  Branches      4047      4049    +2     
=========================================
+ Hits         35010     35033   +23     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@koxudaxi koxudaxi merged commit a660294 into main Jul 7, 2026
62 checks passed
@koxudaxi koxudaxi deleted the perf/builtin-formatter-guards branch July 7, 2026 02:27
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🎉 Released in 0.68.1

This PR is now available in the latest release. See the release notes for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant